DOCUMENT RESUME 

ED 288 921 UD 025 894 



AUTHOR 
TITLE 

INSTITUTION 

Sx'ONS AGENCY 

PUB DATE 

GRANT 

NOTE 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Madden, Nancy A,; Slavin, Robert E. 
Effective Pull-out Programs for Students at Risk. 
Center for Research on Elementary and Middle Schools, 
Baltimore, MD. 

Office of Educational Research and Improvement (ED), 
Washington , DC. 
Apr 87 

OERI-G-86-0006 

34p.; Paper presented at the Annual Meeting of the 
American Educational Research Association (April 
20-24, 1987, Washington, DC). 
Speeches/Conference Papers (150) — Reports - 
Evaluative/Feasibility (142) 

MF01/PC02 Plus Postage. 

Academic Achievement; Class Size; ^Compensatory 
Education; Computer Assisted Instruction; Elementary 
Education; *High Risk Students; *Individuali;e;ed 
Instruction; *Instructional Design; ^Program Design; 
Program Effectiveness; Remedial Instruction 
Education Consolidation Improvement Act Chapter 1 



ABSTRACT 

This paper reviews research on effective pull-out 
programs for elementary students who are at risk for school failure. 
The emphasis is on programs that can be readily replicated by schools 
other than those that developed them. The focus is on programs 
provided to students who have been identified as in need of remedial 
services which are implemented outside of the regular classroom. Th( 
study found that effective pull-out programs for students fall into 
three broad categories: (1) diagnostic-prescriptive programs; (2) 
tutoring programs; and (3) computer-assisted instruction. Successful 
examples of each of these types are described and discussed. The most 
successful models completely adapt instruction to students* unique 
needs and provide plentiful direct instruction appropriate to their 
levels of readiness. Results of the study suggest that the 
achievement of at-risk students can be significantly increased, 
either by making relatively inexpensive but extensive modifications 
in the regular instructional program or by implementing relatively 
expensive but intensive interventions as pull-out programs. It is 
possible that a combination of these strategies would be more 
effective than either one by itself. An extensive list of references 
is included, along with descriptions of successful programs among 
each of the three types. (PS) 
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From the beginnings one of the guiding principles of compensatory educa- 
tion has been that compensatory services must supplement, not supplant, the 
educational programs provided to eligible students. Low achieving students 
in schools which are eligible for Chapter I (formerly Title I) services must 
receive something identifiably "extra." A small army of evaluators located 
in every state check to see that Chapter I services are in fact supplement- 
ing rather than supplanting regular education programs. 

Partly because of the "supplement, not supplant" regulations, schools 
have overwhelmingly relied on pull-out models as a means of providing Chap- 
ter I or Title I services. Most often, students who qualify for compensa- 
tory services are taken out of their regular classrooms for 30-40 minutes of 
remedial instruction in reading and/or mathematics. This arrangement has 
the advantage of making it clear that Chapter I services are supplementary, 
as special personnel and materials are clearly only allocated to identified 
students. In in-class alternatives to pull-out, maintaining the distinction 
between who is served and who is not served is more difficult. At least 
partly for this reason. Chapter I programs hav^e overwhelmingly chosen pull- 
out as the mode of service delivery. A study conducted during the 1981-82 
school year found that pull-out outnumbered in-class models by nine to one 
(Advanced Technology, 1983), and in a more recent study of schools specially 
chosen to represent a variety of service delivery models, fifteen of seven- 
teen elementary schools used pull-out in reading and/or math (Rowan, Guth- 
rie, Lee & Guthrie, 1086). 

Despite the many criticisms of pull-out (see, for example. Glass & Smith, 
1977; Johnston, Allington, & Afflerbach, 1985; Archambault, 1987), pull-out 
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is likely to ranain as a widely-used means of providing compensatory educa- 
tion services. In any case, many have noted previously (e.g., Archambault, 
1987; Kennedy & Biman, 1986), the important issue is not the setting in 
which compensatory services are provided, but the quality of the programs 
provided in the setting. 

This paper reviews research on effective pull-out programs for el-anentary 
students who are at risk for school failure. A companion paper by Slavin & 
Madden (1987) discusses effective classroom programs for the elementary 
grades, and a paper by Karweit (1987) discusses effective preschool and kin- 
dergarten programs. 

Scope of the Review 

The focus of this paper is on programs provided to students who have been 
identified as being in need of remedial services which are implemented out- 
side of the regular classroom. This excludes programs implemented in the 
regular class (which are reviewed by Slavin & Madden, 1987) and self-con- 
tained "replacement" or special education programs in which students are 
assigned to a remedial class for most or all of their school day* The 
emphasis of this review is on programs which could be (or have been) readily 
replicated by schools other than those which developed them. For example, 
many school districts have developed exemplary Chapter I programs (see Gris- 
wo" Cotton, & Hansen, 1986) , but the features of these programs are often 
uniquely adapted to the situations, personnel, and students of the district 
and were not designed to be replicated as such in other districts. Also, a 
few studies have identified variables related to effective Chapter I pro- 
grams (e.g., Cooley & Leinhardt, 1980; Crawford, in press). In contrast to 
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these, the programs emphasized in this review are models which have well- 
specified manuals, materials, training procedures, and other features which 
characterize programs intended for replication by others. 

Review Proc edures 

The procedures used in this review are described in detail by Slavin & 
Madden (1987). Essentially, the review procedure was an adaptation of 
best-evidence synthesis (Slavin, 1986), a method which combines the features 
of meta-analytic and traditional narrative reviews. The initia.^ literature 
search examined many sources, including published and unpublished articles, 
school district reports, and government document.^. Requests for information 
were sent to all of the 116 exemplary compensatory education programs iden- 
tified by Griswold et al. (1986). However, the most useful sources of 
information were reports submitted to the Joint Dissemination Review Panel 
(JDRP), a U.S. Department of Education panel which reviews evaluations of 
programs supported by federal funds. Programs whose effects are certified 
by the JDRP as alid are eligible for funding and dissemination through the 
National Diffusion Network (NDN) . The JDRP submissions are especially use- 
ful because virtually all programs submitted to JDRP are designed to be 
replicated by others and because the JDRP requires data that can be used to 
assess program effects. 

A set of substantive and methodological inclusion criteria were applied 
in deciding which programs to emphasize in this review. These are described 
in detail by Slavin & Madden (1987). In brief, programs had to assess 
effec*:s on standardized reading and/or math scales in studies or at least a 
semester* s duration. Programs had to be compared to matched or randomized 
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control groups, or year-to-year gains of at least five NCE^s had to be 
reported (fall-to-spring NCE gains, which have been found to have serious 
methodological problems, were not considered adequate evidence of program 
effects; see Gabriel, Anderson, Benson^ Gordon, Hill, Pfannenstiel, & Stone- 
hill, 1985). All programs presented data to indicate that they were effec- 
tive with students eligible for Chapter I services or other at-risk stu- 
dents* 



Categories of Effective Pull-Out Models 

The effective pull-out programs for students at risk of school failure 
fell into three broad categories. One, d iagnostic-prescriptive programs , 
covers the great majority of existing Chapter I programs. In this model, 
students identified as being in need of remedial services are carefully ass- 
essed and then instruction ap';^ropriats to their needs is given by a teacher 
in a location separate from the regular classroom. Instruction may be given 
to individuals or to small groups within a pull-out class of roughly three 
to eight students. 

The second category is tutoring programs , in which tutors work one-on-one 
with identified tutees. Tutors may be teachers, paraprof essionals, volun- 
teers, or older students. In the third category, computer-assisted instruc- 
tion (CAI), students work on computers for at least part of their remedial 
reading or math time. It shouxd be noted that many diagnostic-prescriptive 
programs use computers for management (as opposed to instructional) pur- 
poses, but these are not categorized as CAI. 
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Characteristics and Outcomes of Effective Pull-Out Models 
Diagnostic- Prescriptive Programs 

Because of the widespread use of diagnostic-prescriptive pull-out pro- 
graus, the data available on such programs is t^xtensive. Almost all of the 
exemplary Chapter I programs identified by Griswold et al. (1986) used diag- 
nostic-prescriptive models, as did many of the programs certified by the 
JDRP. However, very few of these presented convincing evidence of effec- 
tiveness. In most case£^, these programs were identified as exemplary on the 
basis of fall-to-spring gains in normal curve equivalent scores (NCE's). 
However, most districts presented fall and spring scores for several years, 
and with few exceptions these indicate little or no growth in fall scores or 
spring scores over time. The typical pattern is a fall-to-spring gain of 
8-12 points, followed by a decline over the summer of the same magnitude. 
This cannot be seen as convincing evidence that the programs are not effec- 
tive; it could be that the lack of year-to-year growth in scores is due to 
dropping of students with high scores from the Chapter I lists, or that 
scores increased when new programs were introduced but then remained stable 
during the years for which data were provided, or thare may be other metho- 
dological or substantive factors involved. 



Table 1 Here 



However, there are a few diagnostic-prescriptive programs which do pre- 
sent more convincing evidence of effectiveness* Information on these pro- 
grams is provided in Tabic 1. 
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Only two of the successful diagnostic-prescriptive programs used control 
group designs. One of these^, Project Conquest (JDRP No. 74-12) mostly pro- 
vider; remedial services to groups of six students. However, early in the 
year, students receive one-to-one tutoring until they acquire word percep- 
tion skills. Student^? are carefully assessed by special "reading clini- 
cians," and given individual prescriptions. 

An evaluation of Project Conquest was conducted in low-income, mos-:ly 
black schools in East St. Louis, Illinois. Project Conquest students mad<^ 
greater gains than control students at grade levels f'^om 1-6 on many stand- 
ardized leading measures. However, these data are difficult to interpret 
because no evidence is given that the experimental and control classes were 
initially equivalent. A more sophisticated analysis was conducted using 
fifth grade data and correcting for pretest differences, and this also 
showed a clear advantage for the program. 

One of the exemplary Chapter I programs identified by Griswold et al. 
(1986) used a control group design to evaluate its diagnostic-prescriptive 
pull-out model. This is the Oklahoma City, Oklahoma Chapter I program, a 
classic diagnostic-prescriptive pull-out model. In this program, students 
are assessed in terms of skills and learning styles and then given instruc- 
tion appropriate to their needs, individually or in small groups. Some use 
is made of computer- assisted instruction; the evaluation of this component 
is discussed below in the section on GAI. 

The evaluation of the Oklahoma City diagnostic-prescriptive pull-out pro- 
gram (Kimball, Crawford, & Raia, 1985) involved careful matching on pres- 
cores of students who received Chapter I services with those who did not. 
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The same procedures were followed in two successive years. Results indi- 
cated that Chapter I students gained 3.0 NCE^s more in math than their coun- 
terparts who did not receive Chapter I services. Using the standard devia- 
tion of NCE's of about 12 estimated by Gabriel et al. (1985), this is equi- 
valent to an effect size of +0.25. However, gains in reading were much 
smaller and^ in one year, were not statistically significant (ES=+.12). 

It so happens that Kimball et al. (1985) computed effects of the Oklahoma 
City Chapter I program for 1984-85 three different ways, using fall-to- 
spring, spring-to-spring, and control group designs. It is instructive to 
see the radical differences in the NC^ gains from these three methods, sum- 
marized in the following chart: 

Oklahoma City Chapter I Evaluation, Grades 1-6 
NC^'s Computed Three Ways 

Reading Math 

Fall-to-spring 13.03 17.05 

Spring-to-spring 5.87 8.00 

Kxp' rimental-control 1.90 2.80 

The above chart shows clearly that different evaluation designs lead to 
different conclusions, and counsels extreme caution I'n interpreting results 
of fall-to-spring and spring-to-spring data. 

The Baltimore, Maryland school system also conducted a comparison of stu- 
dents served by Chapter I pull-out programs and matched students eligible 
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for Chapter I but not served. This evaluation found that served and 
unserved students made about the same fall-to-spring gains in reading and 
math scores (Granick, Quigley, Katzenellenbogen, & Richardson, 1984). 

Other than Project Conquest, only one JDRP-approved program appears in 
Table 1, The Diagnostic-Prescriptive Arithmetic Program (JDRP NO. 74-68). 
This model, developed and evaluated in Staten Island, New York, uses a math 
lab approach to remediation of deficits in mathematics. Individualized and 
small group activities keyed to problem areas identified by the Stanford 
Diagnostic Mathematics Test are provided to students. Spring-to-spring 
gains of approximately 10.4 NCE's were made; however, since the Stanford 
Diagnostic Mathematics Test was also used as the posttest, there is a possi- 
bility that this program was essentially teaching the test. 

Two additional programs from the list of effective Chapter I programs 
also presented evidence of effectiveness based on outstanding year-to-year 
gains. One of these is the Lincoln, Nebraska Chapter I program. In this 
program, a computer management system was introduced to help handle diagnos- 
tic tests, assign students to the program, maintain coordination between the 
regular teacher and the Chapter I teacher, monitor student progress, and 
evaluate student success. During the years when the program was imple- 
mented. Chapter I students showed steady gains in spring scores in reading 
and math which appear to be due to the addition of the computer management 
system to the diagnostic-prescriptive pull-out model already in use in the 
district (Weatherl, 1986). 

Another apparently effective Chapter I model cited by Griswold et al. 
(1986) is the Columbia, Missouri Public Schools' Chapter I mathematics pro- 
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gram. This model places considerable emphasis on coordination of instruc- 
tion between Chapter I and regular classroom teachers; forms indicating spe- 
cific objectives students are working on are passed back and forth between 
Chapter I pull-out teachers and regular classroom teachers, and time is set 
aside for Chapter I and regular teachers to meet. Program data indicate 
fall-to- fall gains of 3.7 NCE's in math. 

The evidence from the Chapter I programs listed in Table 1 indicates that 
diagnostic-prescriptive programs can be effective, but their effects do not 
generally appear to be large. However, the almost universal use of fall-to- 
spring gains (or year-to-year gains at best) as the criteria of program 
effects means that we really know little about which programs (or couponents 
of programs) are having important effects on the students they serve. 

Tutoring Programs. 

A wide variety of tutorial programs using tutors ranging from experi- 
enced, specially trained teachers to paraprof essional aides to older, low- 
achieving students have demonstrated considerable effectiveness in improving 
students achievement in reading and math for students in grade 1 through 6. 
These effective progr»*^<? fall essentially into two groups; those that were 
designed as remedial ^ grams and those that were designed as preventative 
programs. Three models for remediation were identified. The two strongest 
models. Training for Turnabout Volunteers (TTV) and School Volunteer Devel- 
opment Project (SVDP), were developed in Dade County, Florida. TTV used 
volunteer junior high school students who took tutoring as an elective class 
to tutor low achieving first through sixth graders in reading and math. 
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Tutors were required to be able to read and compute at the fifth grade 
level, which allowed underachieving students to act as tutors. Tutors 
received considerable training involving a specified structured curriculum 
before they began to tutor, and spent one day out of five in group supervi- 
sion, receiving continuing training in che specific subject matter being 
taught as well as in tutoring skills such a? rewarding the tuteec' suc- 
cesses, refraining from criticizing failures, organizing the work to be pre- 
sented, and so on. The tutorial materials were not programmed. Tutors drew 
from a wide variety of materials deemed to be useful for teaching the needed 
skiJls. The gains made by students working with the well-trained tutors 
were compared to the gains mads by students working with tutors who did not 
receive continuing supervision, but otherwise worked for similar amounts of 
time with similar students using the same kinds of materials. Students 
worked together four days a week for sixteen weeks for forty minutes a day. 

Gains were significantly greater for both tutors and tutees when the 
tutors received continu.lng training. Students tutored by trained tutors in 
math gained .93 standard deviations more than those tutored by untrained 
tutors on the Metropolitan Achievement Test. In reading, the tutees of 
trained tutors gained .51 standard deviations more on the Metropolitan than 
their comparison group* Trained tutors gained .49 standard deviations more 
than their untrained counterparts in math, but did not gain significantly 
more in reading. 

One difficulty with the TTV program was thi'iC it required that an elemen- 
tary and junior high school be physically close to one another so that stu- 
dents could travel back and forth in the time allotted for one class period 
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and still have time to tuto**.. The second program developed in Dade County 
avoided this difficulty by focusing on the recruitment and training of adult 
volunteers as tutors. In the School Volunteer Develc.*flent Project, aduit 
volunteers tutored underachieving first througn sixth graders for a hal/ 
hour a day four or five days a weeV* Tutors were trained prior to tutoring 
in a variety of tutoring skills and use of multimedia materials, and worked 
with the reeding specialist on the skills they were tutoring. Again, a var- 
iety- of materials pertaining to the skills being developed were used. Stu- 
dents were randomly assigned to tutored or untutored groups. Those who 
received tutoring gt ned .50 standard deviatJ jus more in reading and 1.1 
standard deviations more in math than the untutored control students. 

In contrast to the two programs above. Success Controlled Optimal Reading 
Experience (SCORE), the third successful remedial model identified, uses 
highly programmed materials and u very specifically structured tut.oring ses- 
sionv SCORE uses rapid drill and practice in lists of words grouped to 
teach specific decoding skills (short and long vowel sounds, sound blending, 
word patterns such as night, light, fight, and so on). Students who ^^re 
deficient in decoding skills are tutored for 15 minutes a day until tiey 
complete the program, usually a period from tour to six months. 

In th** original evaulation of the program, educationally handicapped stu- 
dents were randomly assigned to a special class for leamxii^s disabled stu- 
detits, to regular class placement with no support, or to regular class 
placement with 15 minutes of tutoring by students in grades 6 to 12. The 
SCORE treatment group exceeded both control groups on measures of word 
recognition and oral reading accuracy. Effect size estimates ranged from .5 



-11- 



ERJC 




to .7 standard deviations. The results were replicated over three succes- 
sive years. The program has been implemented in a wide variety of settings 
with lower middle and upper class populations in urban and rural settings 
using older students, parent volunteers and aides as tutors. Gains in these 
settings have exceeded the gains in the original study in almost every case. 

A limitation of these data lies in the fact that no reading comprehension 
data are provided. The gains seen in word recognition and oral reading 
accuracy are large and important, and would be expected to enable readers to 
pay more attention to comprehending their reading, but there is no guarantee 
that this generalization occurs. Coni7rehension gains would have to be ass- 
essed in any implementation v £ this progr .u. It may be that SCX)RE would 
provide a strong first step in a more comj rogram that explicitly 

addressed comprehension as well. 

Three different programs have addressed the problem of reading failure hy 
attempting to prevent failure at the first grade level. Programmed Tutorial 
Reading (PTR) provides one-to-one tutoring by paraprofessionals to first 
graders in the bottom quartile in reading. The tutoring process is highly 
structured. The tutor uses programmed materials based on the basal series 
used in the school that instruct the tutor in where to start, what to say, 
when to praise, how to respond to a failure in word recognition, and so on. 
Students are essentially taught the words in the lasal using a sight word 
approach, and in addition are taught word analysis and passage comprehension 
skills using the same vocabulary in separate series of lessons. 

Programmed Tutorial Reading has been evaluated against a control group in 
two studies. In the first (Ellson, Harris, and Barber, 1968), students were 
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randomly assigned to receive PTR. directed tutoring, or no tutoring. 
Directed tutoring involved having paraprof essionals tutor students using 
specific materials provided by the teacher that were designed to correspond 
to the instruction going on in the classroom. Tutors in both PTR and 
directed tutoring were trained for about 18 hours prior to the beginning 
tutoring. Results indicated that PTR was more effective than directed 
tutoring and control, but the differences were not large. 

In the second study of PTR (JDRP #74-17) . students recieving fifteen 
minutes of tutoring a day were compared to students receiving no tutoring 
over a one year period. In this study, substantial differences were seen 
between the two groups in favor of the tutored group. 

Two other prevention models used programs designed to improve students' 
skills in specific reading or prereading areas. The Wallach Tutorial Pro- 
gram (Dorval, Wallach, and Wallach, 1978), used paraprof essionals to tutor 
children identified as low in readxness skills for one-half hour per day on 
phoneme identification skills* This ^'raining focused the students on break- 
ing up the sounds in words so that, for instance, beginning sounds could be 
heard, sounded, and represented separately from the rest of the word. Wal- 
lach and Wallach based their program on research showing that disadvantaged 
children were drastically deficient in recognizing phonemes in heard words - 
such as knowing whether house or man starts with the sound "mmmmmmm". The 
tutored children performed at the 56th percentile on the California Test of 
Basic Skills at posttest after having begun at the 17th percentile. The 
control group moved from the 19th percentile to the 35th. The effect size 
for the comparison is .75 standard deviations. 
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Prevention of Learning Disabilities, a program developed by the New York 
University Medical Center^ takes an approach that is somewhat simialr to the 
Wallach and Wallach approach. Students are screened for deficits in sensory 
skills related to reading at the kindergarten or beginning first grade 
level, and those found to be deficient are tutored in those areas in which 
they are low. Auditory discrimination exercises similar to those used by 
Wallach and Wallach form a significant part of this instruction. Students 
were assigned to work on a tutorial basis or in small groups (two to three 
students) with a resource teacher three to five times a week. The program 
is designed as a two-year program, but evaluations after both one and two 
years showed significant gains for the treated group on word recognition 
scales and on a word attack measure. No standardized tests of reading com- 
prehension were used. 

While the evidence of effectiveness for th tutoring programs is very 
impressive and supports the use of such models, one important piece of data 
is lacking. Tutoring models, in particular those using paid tutors and 
which identify themselves as ''preventative," often justify their costs in 
terms of reducing the need for later remediation. However, none of ^:he 
evaluations of preventative tutoring models actually present data on long- 
term maintenance of effects, either on achievement or on assignment to spe- 
cial or remedial education. One preventative tutoring program that has gen- 
erated considerable interest in recent years, Reading Recovery (Clay, 1985; 
Boehnlein, 1987), claims to remove the need for future remediation for most 
of the high-risk first graders it serves, but evidence supporting this is 
lacking as of this writing. 
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Overall, the results of the tutoring studies show this strategy to have 
strong effects on student achievement. VThen older students are used as 
tutors, they also experience important achievement gains; when either peers 
or adult volunteers provide tutoring, the costs of tutoring programs are 
low* 

Computer Assisted Instruction 

Computer assisted instruction (CAI) can be seen as another form of tutor- 
ing in which the "tutor" is a machine rather than a person* Like a human 
tutor, computers can determine students' needs, provide instruction appro- 
priate to those needs, recognize and reinforce student success, and keep 
records of student progress. However, computers are usually less able to 
explain concepts to students, and for this reason CAI programs invariably 
have teachers available to students while they are working on their computer 
lessons. 

Several studies of CAI as a pull-out program for students in need of 
remediation have been conducted, but both the largest number of evaluations 
and the highest quality evaluations have involved the reading and math pro- 
grams developed at Stanford University in the early 1970 's and currently 
disseminated by the Computer Curriculum Corporation, or CCC (Jamison, 
Fletcher, Suppes, & Atkinson, 1976) . Unlike most current CAI programs which 
operate on microcomputers, CCC uses a mainframe with terminals linked to a 
central processing unit by telephone. The computer keeps records of student 
performance levels and progress, and provides students with exercises appro- 
priate to their needs. 
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The most important evaluation of the CCC curriculum itself was conducted 
in the Los Angeles Public Schools by the Educational Testing Service (Rago- 
sta» lSc53). In this four-year longitudinal study, students were randomly 
assigned to receive ten minutes per day of CAI in mathematics, reading, or 
language, as part of a thirty-minute Chapter I pull-out period. Results on 
the CTBS indicated substantial effects of CAI on math computations, which 
increased from an effect size of .36 in the first year to .72 by the third 
year of intervention. Effects on concepts and applications scales were 
smaller and non-signiiicant. In reading, positive effects were found for 
vocabulary and comprehension scales after one year (ES=+.25 and +.23, 
respectively), but while the vocabulary effects increased to +.59 by the 
third year, comprehension effects actually favored the control group by that 
time (ES=:-.24). Effects in language mechanics were at about one-quarter of 
a standard deviation all three years, but were small and generally non-sig- 
nificant on language expression and spelling scales. 

Two JDRP- approved projects which used the CCC curriculum also present 
convincing evidence of effectiveness. One, a program evaluated in Lafayette 
Parish, Louisiana, called Title I Mathematics Laboratory with Computer 
Assisted Instruction (JDRP No. 82-46), randomly assigned students to regular 
Title I pull-out or to a combination of the regular pull-out model with ten 
minutes of CAI in mathen.atics. Effects on the CTBS Total Math Scale were 
modest, only +.19, but the extraordinary quality of the experiment makes the 
effect credible. Another JDRP program based on the CCC reading materials is 
the Merrimack (Massachusetts) Education Center (JDRP No. 82-34), a program 
which supplements ten minutes per day of CAI with tutorial and small group 
instruction from the teacher. A study with random assignment of students to 
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CAI and control treatments found positive effects averaging 40% of s stan- 
dard deviation on the MAT reading scale. 

One interesting CAI program which does not use CCC materials is Basic 
Literacy through Microcomputers, a model developed and evaluated in Salt 
Lake City, Utah. This program uses either electric typewriters or computers 
to supplement the teacher's instruction by giving students opportunities to 
practice and apply ohonics skills by typing words, sentences, and stories. 
The evaluation of this program involved comparison classes in gains on 
standardized tests. Differences at grades 1 and 3 favored the experimental 
group, but substantial pretest differences at grade 3 make these results 
inconclusive. Ironically, the grade 1 experiment, which found substantial 
positive effects on California Achievement Test Total Reading scores 
(ES=+.58), used typewriters rather than computers. 

Overall, results for the CAI programs (especially CCC) are well-estab- 
lished and positive, though in the best-controlled studies they are usually 
modest in magnitude and appear more frequently on basic skills than on high- 
er-order skills. Since the costs of CAI can be very high (see Ragosta, 
1983), this approach can be compared to adult tutoring, which tends to have 
larger effects in studies of similar methodological quality. 
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Conclusions 



In a companion paper to the present article. Slavin & Madden (1987) 
reviewed research on effective classroom programs for students at risk for 
school failure. They found that the most consistently successful classroom 
models were continuous-progress programs in which students are taught in 
skill-level groups and proceed through a hierarchical set of skills, &nd 
cooperative learning programs in which students also receive instruction at 
their appropriate levels but then practice skills in mixed-ability learning 
teams. On the basis of this and other evidence, Slavin & Madden (1987) con- 
cluded that effective programs for students at risk balanced adjustment of 
instructional approaches to meet students' unique needs with provision of 
adequate direct instruction. In addition, effective classroom programs pro- 
vide frequent assessment of student progress through a well-specified, hier- 
archical set of skills. 

The examination of effective pull-out programs conducted in the present 
paper provides further support for these conclusions. The most successful 
models, tutoring and CAI, completely adapt instruction to students' unique 
needs and provide plentiful direct instruction appropriate to students' lev- 
els of readiness. Diagnostic-prescriptive pull-out models, which have gen- 
erally been less successful than tutoring or CAI, also carefully assess stu- 
dents' needs and adapt instruction to these needs, but often suffer from one 
of tvro problems. 

Chapter I pull-out teachers often work with individual students, leaving 
others in the group to spend much time working on worksheets, which may be 
of relatively little value. Alternatively, Chapter I teachers may present 
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lessons to heterogeneous groups of students which do not adapt to their 
individual needs. In either case^ the instruction provided in traditional 
diagnostic-prescriptive pull-out programs may not be markedly better than 
that provided in the regular classroom. If so, it is unrealistic to expect 
that 30-45 extra minutes of instruction will make a substantial difference 
in achievement. Both the tutoring and the CAI studies suggest that inten- 
sive interventions are needed to make a substantial difference in a pull-out 
program. 

Taken together, the conclusions of the present paper and of Slavin & Mad- 
den (1987) suggest that the ^cliievement of at-risk students can be signifi- 
cantly increased, either by making relatively inexpensive but extensive 
modifications in the regular instructional program or by implementing rela- 
tively expensive but intensive interventions as pull-out programs. It is 
possible that a combination of these strategies would be more effective than 
either one by itself. 

There is much more we need to know about effective programs for students 
at risk. Research on diagnostr.c-prescriptive models is generally of low 
quality; it may be that certain forms of diagnostic-prescriptive models 
would be highly effective in well-designed experiments, but much research 
and development is needed to establish what these forms might be and how 
much differsnce they could make in student achievement. The tutoring and 
CAI studies are of much higher metnodological quality, but except in cases 
where peer tutoring or volunteer programs are practical, these are expensive 
interventions. For practical as well as theoretical reasons, then, there is 
a need to identify the elements of tutoring and CAI that account for their 
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effects, so that perhaps these same principles ccul-i be applied in a less 
expensive form. 

We can make an important difference in the achievement of students at 
risk for school failure. Of that there is no doubt. All that is in doubt 
is our willingness as a society to support continued research and develop- 
ment of effective, replicable models and to support the systematic use of 
the methods research has already identified. 
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Table 1: Effective Diagnostic-Prescriptive Pullout Programs 



PROGRAM AND DESCRIPTION 



SAMPLE 



DESIGN 



EFFECTS 



EFFECT SIZE 



Programs Evaluated Using Control Groups 



Project Conquest 
East St. Louis, 197A. 
(JDRP #74-12) 
Individual and small 
group instruction 
provided outside of 
class to r<anediate 
specific deficits 
identifed by indivi- 
dual assessment* 
Extensive inservice. 



Grades 1-6, reading. 
Students performing 
below grade level. 



Comparison group 
identified (cri- 
teria not clear); 
IQ's slightly 
lower for compari- 
son group. 



Treatment group gains in each grade 
exceeded comparison group gains 
significantly on standardized 
reading tests. No evidence of 
other adoptions. 



Cannot be 
estimated. 



Oklahoma City, OK 
Chapter 1 Program. 
(Kimball, Crawford, 
and Raia, 1985) 
District-wide 
management objectives 
used to coordinate 
regulaL class and 
Chapter 1 instruction. 



Grades 1-8, reading 
and math. 

Chapter 1 students. 



Control group 
selected from 
Chapter 1 eligible 
students not served. 



California Achievement Test 

Math- trt gained 3.0 NCE's more ES=.25 

than control 
Reading- trt gained 1.45 NCE's ES=.12 

more than control 



f 
( 
f 
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Table U continued 



PROGRAM AND DESCRIPTION 



SAMPLE 



DESIGN 



Programs Evaluated Using Year-to-Year Gains 



Lincoln* Nebraska 
Chapter I Manegcfflent 
6nd Coordination 
Proj ect. 

Developed district 
l5."V€l objectives and a 
monitoring rystem. 
Coaputeri< used for 
management fo data. 
Day-to-day coordi- 
nation between Chapter 1 
and regular class. 



Grades K-6, reading 
and math. 

Chapter 1 students. 



No control group. 
Spring-to- spring 
gains measured. 



Diagnost i c-Presc r iptiv e 
Arithmetic Program, 1972. 
Staten Island. NY. 
Math lab approach 
emphasizes hands-on 
experience* manipulatives. 



Grades 3-5, math. 
Students two years 
below grade level, 
level . 



No control group. 
Spring-to-spring 
gains measured 
(student level) . 



Columbia* l^l. 
Chapter I Math Program. 
Emphasizes coordination 
between Chapter I and 
regular classroom* as 
well as Ui;e of 
manipulatives in math 
instruction. 



Grades 2-6, math. 
Chapter I students. 



No control group. 
Fall-to-fall scores 
for successive co- 
horts. 



f 
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EFFECTS 



EFFECT SIZE 



Spring-to-spring gft:ns of A-5 Not estimated. 

NCE's in reading ana r ? m 
math. System-wide yc*- ^-t-o-year 
continuing gains are shown. 



Stanford Diagnostic Arithmetic Not estimated. 

Test. Spring-to-spring gains 
of 10.4 NCE's. 



Iowa Test of Basic Skills. Not estimated. 

NCE gains fall-to-fall of 
8.7 NCE's. 
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Table 


2: Effective Tutoring Programs 






t 


PROGRAM AND DESCRIPTION 


SAMPLE 


DESIGN 


EFFECTS 


EFFECT SIZE 






.Programs Evaluated Using 


Control Groups 












A. Remedial Programs 














Training for Turnabout 


Tutors 7-9 graders 


Compared trained 


Tutees 








^ Volunteers. 


Tutees 1-6 graders 


V3. untrained 


Metropolitan Ach. Test (Math): 








(JDRP #81-11) 


Five schools in 


tutors. Non- 


Tutees having trained tutors 


ES=.93 






Cross-age tutoring in 


Miami (Dade Cty. ) 


equivalent but 


(f*=62) gained significantly more 






' reading and math. Tutors 


Tutored in reading 


unbiased groups. 


than those having untrained 








trained with pre- service 


and math. 


Duration of 


tutors (N=62). 








and inservice classes. 




tutoring - o-^e 


Metropolitan A'^hievement Test 








Not programmed material. 




school year. 


(Reading): Tutees having trained 








Tutored AO min./day 






tutors (N=5l) gained signifi- 








four days/week. In- 






cantly more than those having 


ES=.51 






service training on 






untrained tutors (N=51) 






fifth day. 














Cost minimal - 1 (part- 






Tutors 








time) paid staff per 






MAT (Math) trained tutors > 


ES=.49 






school . 






untrained 












MAT (Reading) trained tutors > 














untrained 


ES=.08 






School Volunteer Devel- 


Reading and math. 


Random assignment 


Metropolitan Achievement Test 








opment Proj ect 


Grades 2-6 


to tutored or 


Reading tutored (N=119) > 








(JDRP #75-79) 




untutored groups. 


untutored (N=117) by 


ES=.50 






Community Volunteer Tutors. 


Duration of 


1 GE gain 






Tutored 2-4 hrs./week/ 




tutoring - one 


Math tutored (N=119) > 








volunteer. Each student 




school year 


untutored (N=117) by 








tutored 1/2 hr. k days/ 






,85 GE gain 


F.S=1.1 






week minimum. 














Cost minimal - 3 paid 














staff per district. 














Success Controlled 


Grades 1-6 


Randomly assigned - 


Tutored (N=6l) > untutored (N=58) 


WRAT ES=.5-.7 






Optimal Reading 


Reading 


2 controls repli- 


on word recognition (WRAT/ Gil more) 


Gilmore ES=.5 






Experience (SCORE) 




cated over 3 yrs. 


NO TESTS OF COMPREHENSION 








(JDRP #80-42) 














San Francisco* CA 














Each student tutored 














15 min./day by older 














students or adult 














volunteers. 










Or 
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Table 2. continued 



PROGRAM AND DESCRIPTION 



SAMPLE 



DESIGN 



EFFECTS 



EFFECT SIZE 



B. Preventative Programs 

Programmed Tutorial 
Reading (Study 1) 
Fannington, Utah (72-73) 
(JDRP #7A-17) 
Each student tutored 
15 min./day by para- 
professionals or older 
students. 



Grade 1, reading, 
bottom quartile 



Matched exp./ 
control students 
in pairs - no 
pretest differences 
Duration: 1 year 



Gates McGinty Reading Scores 

Tutored (N=33) > untutored (N=33) 

Replicated using non-standardized 
tests in several other districts. 



ES=.65 

vocabulary 

ES=.A1 

comprehension 
ES= ,50 
basal 

comprehension 
measure 



Programmed Tutorial 
Reading (Study 2) 
(Ellson, Harris, Barber, 
1968) 

Each student tutored 
15 min./day by para- 
professionals. 
Programmed tutoring 
compared to directed 
tutoring (comparable to 
the regular classroom 
activities done with 
the individual student) 



Grade J , reading, 
selected from entire 
rangf. of first graders 
in t.he class, 

N=:A3 in each group 



Sixteen schools in 
inner city. Random 
assignment to ex- 
perimental groups 
then matched con- 
trol selected from 
same class, Exp. 
trts. were; 

1 or 2 sessions programmed 
1 or 2 sessions directed 
tutoring 



Programmed tutoring > directed 
tutoring on Stanford Ach. Test 
Total Reading 

Tutoring of any kind > no 
tutoring on criterion referenced 
measures only 



ES=,10 



Prevention of Learning 
Disabilities - New Yotk 
(JDRP #79-33) 
Each student tutored 3-5 
times/week for 30 min, 
by a resource teacher. 



Grades 1 & 2 
Students with low 
readiness scores. 



Rardcnly assigned 
to test or control 

Measured at end of 
1st and 2nd grade 



Tutored (N=87) > untutored (N=39) 

WRAT ES=1.00 

Woodcock Word Attack ES=2.00 

Woodcock Word Recognition ES=1.00 



Wallach Tutorial Program 
(Dorval, Wallach and 
Wallach 1978) 
Half-hour per day 
28 hours total 
Tutored by para- 
professionals. 
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Grade 1 - students 
below 40% on MAT 
TRT N=20 

One control group from 
same school (N^20) 
One control group from 
different school (N=18) 
Nine student control 
groups received 
extra help from an 
aide. 



Not clear how 
students were 
assigned to treat- 
ments but groups 
are equivalent at 
pretest. Treat- 
ments were 
tutoring vs, no 
tutoring vs, 
time with aide. 



Spache Word Rec. 
Spache Passages 
CTBS Total Rdg, 



tutored > control 
1. Lored > control 
tutored > control 



Diff=1.3 GE 
ES=,75 
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Table 3: Effective Computer- Assisted Instruction Programs 



PROGRAM AND DESCRIPTION 



SAMPLE 



DESIGN 



EFFECTS 



EFFECT SIZE 



Programs Evaluated Using Control Groups 

Computer Curriculum Corp* 
Drill and practice curr- 
icula in reading* math 
and language arts 



Q^11/1«T 1 * T firs Art 

ouuay X* IjOS Angeies 


nam » ur aaes i o 


ocAuparcu lo ranuuniiy 


-1 f ot-n-i A T£»o*- r»f Rflc-1 r *?lf -i 11c ( (TTRS) 
V»€lX 1 X Ul n X o XeSU ox faoX^ ul^XXXo \\jl.L/iJ/ 


vr 1 


vr 2 


vr 3 


Unified School District 


Reading, 3-6 


selected controls. 


Math Computations 


+ .36 


+ .56 


+ .72 


4 schools (Ragosta» 


Language, 3-6 


Groups studied 


Math Concepts 


-.02 


+ .12 


+ .09 










+ .03 


+ .12 


+ .26 






4 years • 


PpflHino VomViiilnrv 


+ .25 


+ .17 


+ .58 






Reading Comprehensions 


+ .23 


-.01 


-.24 








St)P 1 lino 


+ .14 


+ .05 


+ .14 








T.nnoimc^p Mpthnni re 


+ .22 


+ .27 


+ .25 








Language Expression 


+ .11 


+ .05 


+ .23 


Study 2: Lafayette 


Math, Grades 3-6 


Compared to students 


CTBS Total Math: CAI > control 




ES= 


= .19 


Parish, Title I Math 


Chapter 1 students 


receiving Chapter I 










(JDRP #82-46) 




pull out* Randomly 














assigned* 










Study 3: Merrimack 


Reading, Grades 2-9 


Compared to students 


Metropolitan Achievement Test, 




ES- 


= .40 


Education Center 


Chapter 1 students 


receiving Chapter I 


Reading: CAI > control 








(JDRP #82-34) 




pullout. Randomly 














assigned* 










Basic Literacy Through 


Reading, Grade 1 


Compared to students 


California Achievement Test, Reading: 




E? 


.58 


Microcomputers (micro- 


Chapter 1 students 


receiving no add- 


Treated students > control 








computers or type- 




itional time. 










writers used) 




Randomly assigned. 










(JDRP #84-14) 
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